Papers with short-context models
Token Weighting for Long-Range Language Modeling (2025.findings-naacl)
Copied to clipboard
| Challenge: | Many applications of large language models (LLMs) require long-context understanding, but models still struggle with such tasks. |
| Approach: | They propose token-weighting schemes that assign different weights to each training token in the loss, generalizing existing works. |
| Outcome: | The proposed methods compare confidences of a long-context and short-concept model and show that non-uniform loss weights improve the long-constability of LLMs. |
Adapting Pretrained Text-to-Text Models for Long Text Sequences (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing short-context models are limited in their domain coverage and can be used for long-sequence inputs. |
| Approach: | They propose to replace full attention in transformers with pooling-augmented blockwise attention and pretrain the model with a masked-span prediction task with spans of varying lengths. |
| Outcome: | The proposed model outperforms existing models on long-sequence summarization tasks and achieves competitive performance on long document corpora. |